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GEOMETRIC,  ALGEBRAIC  AND  TOPOLOGICAL  STRUCTURE 
FOR  SIGNAL  AND  IMAGE  PROCESSING 


AFOSR  GRANT  FA9550-04- 1-0094 
FINAL  REPORT 

PRINCIPAL  INVESTIGATOR:  MICHAEL  KIRBY 
DEPARTMENT  OF  MATHEMATICS 
COLORADO  STATE  UNIVERSITY,  FORT  COLLINS,  CO  80523. 

1.  Objectives.  Military  information  collected  from  streaming  video  of  real-time 
unmanned  aerial  reconnaissance,  battlefield  operations  or  from  multi-spectral  satel¬ 
lite  imagery,  for  example,  require  intelligent  processing  by  analysts  to  reduce  response 
time  and  increase  accuracy  of  interpretation.  As  the  National  Defense  becomes  in¬ 
creasingly  dependent  on  evaluation  of  data  produced  by  state-of-the  art  technology 
it  is  a  matter  of  critical  strategic  importance  to  develop  the  associated  basic  mathe¬ 
matical  theory  to  assist  in  high  data  rate  information  processing  and  analysis. 

In  addition  to  exploitable  statistical  features,  signals  and  images  possess  infor¬ 
mation,  or  mathematical  coherent  structure  reflected,  for  example,  by  the  existence 
of  low-dimensional  manifolds.  This  work  concerns  the  development  of  a  range  of 
fundamental  mathematically  based  techniques  drawing  from  algebra,  geometry  and 
topology  that  will  be  applied  to  efficiently  represent,  process  and  analyze  a  wide  range 
of  data  sources  under  varying  conditions  to  ultimately  provide  a  better  understanding 
of  the  data  content. 

2.  Final  Status  of  Effort.  Below  we  highlight  some  of  the  main  achievements 
of  the  research. 

2.1.  Recognition  of  Human  Faces  at  Ultra  Low-Resolution.  Recent  work 
has  established  that  digital  images  of  a  human  face,  collected  under  various  illumina¬ 
tion  conditions,  contain  discriminatory  information  that  can  be  used  in  classification. 
In  this  paper  we  demonstrate  that  sufficient  discriminatory  information  persists  at 
ultra-  low  resolution  to  enable  a  computer  to  recognize  specific  human  faces  in  set¬ 
tings  beyond  human  capabilities.  For  instance,  we  utilized  the  Haar  wavelet  to  modify 
a  collection  of  images  to  emulate  pictures  from  a  25-  pixel  camera.  From  these  modi¬ 
fied  images,  a  low-resolution  illumination  space  was  constructed  for  each  individual  in 
the  CMU-PIE  database.  Each  illumination  space  was  then  interpreted  as  a  point  on 
a  Grassmann  manifold.  Classification  that  exploited  the  geometry  on  this  manifold 
yielded  error-free  classification  rates  for  this  data  set.  This  suggests  the  general  utility 
of  a  low-resolution  illumination  camera  for  set-based  image  recognition  problems. 

2.2.  Principal  Angles  Separate  Subject  Illumination  Spaces.  The  theory 
of  illumination  cones  is  well  developed,  and  has  been  tested  extensively  on  the  Yale 
Face  Database  B  (YDB)  and  CMU-PIE  (PIE)  data  sets.  This  paper  shows  that  if  face 
recognition  under  varying  illumination  is  cast  as  a  problem  of  matching  sets  of  images 
to  sets  of  images,  then  the  minimal  principal  angle  between  subspaces  is  sufficient  to 
perfectly  separate  matching  pairs  of  image  sets  from  non-matching  pairs  of  image  sets 
sampled  from  YDB  and  PIE.  This  is  true  even  for  subspaces  estimated  from  as  few  as 
six  images,  and  when  one  of  the  subspaces  is  estimated  from  as  few  as  three  images, 
if  the  second  subspace  is  estimated  from  a  larger  set  (ten  or  more).  This  suggests  the 
variation  under  illumination  may  be  thought  of  as  useful  discriminating  information 
rather  than  unwanted  noise. 
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2.3.  Examples  of  Set-to-Set  Image  Classification.  Our  initial  attraction 
to  the  set-to-set  paradigm  arose  in  the  multi-class  problem  where  the  sets  where 
generated  by  varying  illumination  over  a  person.  We  also  found  that  different  types 
of  sets  also  afforded  interesting  classification  results.  For  example,  a  collection  of 
images  of  dogs  can  be  compared  to  a  collection  of  images  of  cats.  We  found  that 
the  label  of  a  set  of  images,  i.e.,  whether  the  set  consisted  of  dogs  or  cats  could 
be  determined  with  high  reliability.  Further  experiments  showed  that  the  gender  of 
a  set  of  images  (the  images  would  either  be  all  female  or  all  male)  could  also  be 
determined  with  high  reliability.  This  research  is  being  extended  to  images  provided 
by  the  forest  service  where  they  are  interested  in  whether  the  set  consists  of  managed 
or  non- managed  forests. 

2.4.  Variations  in  Pose  and  Illumination.  We  present  a  face  recognition 
method  using  multiple  images  where  pose  and  illumination  are  uncontrolled.  The 
set-to-set  framework  can  be  utilized  whenever  multiple  images  are  available  for  both 
gallery  and  probe  subjects.  We  can  then  transform  the  set-to-set  classification  problem 
as  a  geometric  one  by  realizing  the  linear  span  of  the  images  in  a  given  resolution  as  a 
point  on  the  Grassmann  manifold  where  various  metrics  can  be  used  to  quantify  the 
closeness  of  the  identities.  Contrary  to  a  common  practice,  we  will  not  normalize  for 
variations  in  pose  and  illumination,  hence  showing  the  effectiveness  of  the  set-to-set 
method  when  the  classification  is  done  on  the  Grassmann  manifold.  This  algorithm 
exploits  the  geometry  of  the  data  set  such  that  no  training  phase  is  required  and  may 
be  executed  in  parallel  across  large  data  sets.  We  present  empirical  results  of  this 
algorithm  on  the  CMUPIE  Database  and  the  Extended  Yale  Face  Database  B,  each 
consisting  of  67  and  28  subjects,  respectively. 

2.5.  Short  Time  Principal  Component  Analysis.  Experiments 
demonstrated  that  EEG  representations  based  on  short-time  PCA  can  be  classified 
by  simple  linear  discriminant  analysis  (LDA)  with  an  accuracy  of  about  80the  sen¬ 
sitivity  of  this  result  to  the  values  of  the  representation  parameters  suggests  that 
time-embedding  is  necessary  for  good  performance  and  that  the  discriminatory  infor¬ 
mation  is  not  isolated  to  a  small  number  of  basis  vectors.  Analysis  of  the  classifiers 
weights  revealed  that  short-time  PCA  basis  vectors  late  in  the  sequence  play  signif¬ 
icant  roles,  suggesting  that  the  low-variance  activity  represented  by  these  vectors  is 
strongly  related  to  the  mental  task.  This  hypothesis  warrants  further  study.  Informa¬ 
tion  gleaned  from  analyses  can  be  used  to  select  subsets  of  features  to  greatly  reduce 
the  dimensionality  of  the  data  and  possibly  improve  the  generalization  performance  of 
the  classifiers.  Extending  this  analysis  to  consider  the  time  course  of  significant  elec¬ 
trodes  and  basis  vector  directions  could  lead  to  hypotheses  of  the  underlying  cognitive 
activity. 

2.6.  A  Pseudo-Isometric  Embedding  Algorithm.  We  present  a  new  Whit¬ 
ney  like  algorithm  for  finding  a  low  dimensional  pseudoisometric  embedding  of  a 
sampled  Riemannian  manifold.  Tangent  spaces  on  the  manifold  are  estimated  from 
the  data  and  then  projected  using  a  criterion  that  ensures  optimal  smoothness  of  the 
inverse.  This  short  projection  is  not  isometric  but  can  be  made  to  be  approximately 
isometric  by  determining  an  appropriate  global  lengthening  transformation  in  the  em¬ 
bedded  space.  We  illustrate  the  application  of  this  algorithm  on  numerically  obtained 
solutions  of  the  Kuramoto-Sivashinksy  partial  differential  equation. 

2.7.  Mapping  the  Identity  for  Classification  in  Nonlinear  Spaces.  Con¬ 
structing  functions  from  data  for  mapping  the  identity  provides  a  natural  means  for 
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performing  classification  in  high  dimensions.  We  illustrate  this  idea  with  the  ap¬ 
plication  of  the  multivariate  state  estimation  technique  (MSET)  a  method  normally 
considered  in  the  context  of  failure  prediction  and  fault  detection.  This  approach 
provides  a  fast  alternative  to  other  approaches  that  depend  on  nonlinear  optimization 
algorithms  such  as  nonlinear  bottleneck  neural  networks.  We  propose  the  application 
of  MSET  to  the  image  or  video  classification  problem  and  present  the  results  of  a  face 
recognition  experiment  under  variations  in  illumination. 

3.  Summary  of  Accomplishments.  Over  the  course  of  this  award  twenty  pa¬ 
pers  were  written,  nine  talks  were  presented  and  eight  disclosures  were  made  resulting 
in  three  patent  applications. 

4.  Personnel  Supported.  The  following  personnel  were  supported  by  this 
award: 

•  Michael  Kirby,  PI. 

•  Silvia  Osnaga,  Graduate  Research  Assist antship. 

•  Arta  Jamshidi,  Graduate  Research  Assistantship.  He  is  scheduled  to  graduate 
in  spring  2008. 

5.  Technical  Publications.  Over  the  period  of  the  grant  twenty  papers  were 
written  14  of  which  have  appeared  and  6  of  which  are  submitted. 

1.  A.  Jamshidi  and  M.  Kirby,  Skew  Radial  Basis  Functions ,  submitted  2007. 

2.  Yui  Man  Lui  and  J.  Ross  Beveridge  and  Bruce  A.  Draper  and  Michael  Kirby, 
Image- Set  Matching  using  Canonical  Correlation  Analysis ,  submitted  2007. 

3.  J.R.  Beveridge,  Jen-Mei  Chang,  Michael  Kirby,  and  Chris  Peterson,  Feature 
Patch  Illumination  Spaces  and  Karcher  Compression  for  Face  Recognition  via 
Grassmannians ,  submitted  2007. 

4.  J.R.  Beveridge,  Bruce  Draper,  Jen-Mei  Chang,  Michael  Kirby,  Holger  Kley 
and  Chris  Peterson,  Principal  Angles  Separate  Subject  Illumination  Spaces  in 
YDB  and  CM U- PIE ,  (submitted  2007). 

5.  David  W.  Dreisigmeyer  and  Michael  Kirby,  A  Pseudo-Isometric  Embedding 
Algorithm  (submitted  2007). 

6.  David  W.  Dreisigmeyer  and  Michael  Kirby,  Mapping  the  Identity  for  Classi¬ 
fication  in  Nonlinear  Spaces,  (submitted  2007). 

7.  Jen-Mei  Chang,  M.  Kirby,  H.  Kley  and  C.  Peterson,  R.  Beveridge,  B.  Draper, 
Recognition  of  Digital  Images  of  the  Human  Face  at  Ultra  Low  Resolution  via 
Illumination  Spaces ,  In  ACCV  ’07,  part  2  (8t/l  Asian  Conference  on  Computer 
Vision,  Tokyo,  Japan)  in  Springer  Lecture  Notes  in  Computer  Science,  Vol. 
4844,  pg  733-743,  (2007). 

8.  Jen-Mei  Chang,  Michael  Kirby  and  Chris  Peterson,  Set-to-Set  Face  Recogni¬ 
tion  Under  Variations  in  Pose  and  Illumination ,  2007  Biometrics  Symposium, 
Baltimore,  MD,  September,  2007. 

9.  A.  Jamshidi  and  M.  Kirby,  A  spatio-temporal  criterion  for  automatic  model 
order  determination,  SIAM  Journal  of  Scientific  Computing,  Vol.  29,  941, 
May,  2007. 

10.  Jen-Mei  Chang,  Michael  Kirby,  Holger  Kley,  Chris  Peterson,  J.R.  Beveridge 
and  Bruce  Draper,  Examples  of  Set-to-Set  Image  Classification ,  In:  Seventh 
International  Conference  on  Mathematics  in  Signal  Processing  Conference  Di¬ 
gest,  The  Royal  Agricultural  College,  Cirencester,  Institute  for  Mathematics 
and  its  Applications,  December,  2006,  pp.  102-105. 

11.  Yue  Qiao,  Larry  Ernst  and  M.  Kirby,  Developing  a  Computational  Radial 
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Basis  Function  (RBF)  Architecture  for  Nonlinear  Scattered  Color  Data ,  Pro¬ 
ceedings  NIP22  International  Conference  on  Digital  Printing  Technologies, 
Sept.  2006. 

12.  C.  Anderson,  M.  Kirby,  J.N.  Knight,  Classification  of  Time  Embedded  EEG 
Using  Short  Time  Principal  Component  Analysis,  (Book  Chapter  to  appear) 
In  Towards  Brain  Computer  Interfacing,  edited  by  G.  Dornhege,  J.  del  R. 
Millan,  T.  Hinterberger,  D.J.  McFarland,  and  K.R.  Muller,  The  MIT  Press, 
2006. 

13.  J.M.  Chang,  R.  Beveridge,  B.  Draper,  M.  Kirby,  H.  Kley  and  C.  Peterson, 
Illumination  Face  Spaces  are  Idiosyncratic ,  IPCV’06,  Vol  2.,  390-396,  2006, 
CSREA  Press. 

14.  Anderson,  C.W.,  Knight,  J.N.,  O’Connor,  T.,  Kirby,  M.J.,  and  Sokolov,  A. 
(2006)  Geomeric  Subspace  Methods  and  Time-Delay  Embedding  for  EEG 
Artifact  Removal  and  Classification,  IEEE  Transactions  on  Neural  Systems 
and  Rehabilitation  Engineering ,  vol.  14,  no.  2,  pp.  142-146,  June  2006. 

15.  A.  A.  Jamshidi  and  M.  J.  Kirby,  Examples  of  Compactly  Supported  Functions 
for  Radial  Basis  Approximations ,  Proceedings  of  The  2006  International  Con¬ 
ference  on  Machine  learning;  Models,  Technologies  and  Applications,  Editors 
H.  R.  Arabnia,  E.  Kozerenko  and  S.  Shaumy,  155-160,  2006. 

16.  Anderson,  C.W.,  Knight,  J.N.,  Kirby,  M.J.  (2005)  An  Inexpensive  Brain- 
Computer  Interface  Based  on  Spatial  and  Temporal  Analysis  of  EEG.  Pro¬ 
ceedings  of  HCI  International,  (HCI-I)  2005,  Las  Vegas,  NV,  (CD-ROM). 

17.  David  A.  Peterson,  James  N.  Knight,  Michael  J.  Kirby,  Charles  W.  Anderson 
and  Michael  H.  Thaut,  Feature  selection  and  blind  source  separation  in  an 
EEG-based  brain-computer  interface,  EURASIP  Journal  on  Applied  Signal 
Processing,  vol.  2005,  issue  19,  pp.  3128-3140. 

18.  D.  Broomhead  and  M.  Kirby,  Large  Dimensionality  Reduction  using  Secant- 
based  Projection  Methods:  The  Induced  Dynamics  in  Projected  Systems ,  Non¬ 
linear  Dynamics  41(1-3)  (August  2005),  pp.  47-67. 

19.  M.  Kirby,  Nonlinear  Signal  Processing,  Encyclopedia  of  Nonlinear  Science, 
ed.  Alwyn  Scott.  New  York  and  London:  Routledge,  2004. 

20.  A.  Fox,  M.  Kirby,  M.  Montgomery,  J.  Persing,  A  Comparison  of  Optimal 
Low  Dimensional  Projections  of  a  Hurricane  Simulation.  In:  Dynamics  and 
Bifurcation  of  Patterns  in  Dissipative  Systems,  G.  Dangelmayr  and  I.  Oprea 
(eds.),  World  Scientific  Series  on  Nonlinear  Science,  Vol.  12,  pages  292-308, 
World  Scientific,  Singapore,  2004 
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6.  Interactions/Transitions. 

6.1.  Presentations.  Over  the  course  of  the  award  nine  presentations  of  the 
results  of  this  project  were  made.  The  UCLA  workshop  invited  40  experts  to  present 
lectures  to  a  large  audience  over  a  one  week  period. 

1.  Some  Mathematical  Ideas  for  Attacking  the  Brain  Computer  Interface  Prob¬ 
lem,  3rd  Annual  Intermountain/Southwest  Conference  on  Industrial  and  In¬ 
terdisciplinary  Mathematics,  Arizona  State  University,  February  2004. 

2.  Dimensionality  Reduction  using  Secant-based  Projection  Methods ,  MGA  Work-| 
shop  III:  Multiscale  structures  in  the  analysis  of  High-Dimensional  Data, 
IPAM,  UCLA  October  2004. 

3.  Reduction  of  Dynamical  Systems  using  Secant-based  Projection  Methods:  The 
Induced  Dynamics  in  Projected  Systems ,  IMA  Signal  Processing  Conference 
2004,  Cirencester,  UK. 

4.  Face  Recognition,  ISTEC  Symposium,  Colorado  State  University,  2005. 

5.  New  Techniques  and  Results  for  the  Brain  Computer  Interface  Problem,  In¬ 
vited  Plenary  Talk,  AFOSR  Image  Processing  Review,  University  of  North 
Carolina  State,  May,  2005. 

6.  Discrete  and  Continuous  Wavelet  Analysis  for  Multiscale  Ecological  Analysis, 
PRIMES  Lecture  and  Lab  Module,  Colorado  State  University,  January,  2006. 

7.  Examples  of  Set-to-Set  Face  Recognition,  Conference  on  Mathematics  in  Sig¬ 
nal  Processing,  Institute  for  Mathematics  and  its  Applications,  Royal  Agri¬ 
cultural  College,  Cirencester,  U.K.,  December  2006. 

8.  Some  Open  Problems  in  the  Analysis  of  Large  Data  Sets,  Workshop  on  New 
Directions  in  Complex  Data  Analysis,  Breckenridge,  CO,  3/07. 

9.  Pattern  Recognition  Using  Sets  of  Observations,  AFOSR  Sensing  Program 
Review,  Harvard  University  6/07. 

6.2.  Transitions.  Related  activities  that  benefit  from  this  effort.  The  MSPA- 
MCS  NSF  award  is  particularly  noteworthy  as  it  permitted  an  expansion  of  the  col¬ 
laboration  to  include  faculty  from  the  Department  of  Computer  Science  at  Colorado 
State  University. 

1.  CO-Principal  Investigator,  CMG:  Analysis  of  Transport,  Mixing,  and  Coher¬ 
ent  Structures  in  Hurricane  Intensity,  NSF  ATM-530884,  Funding  10/1/05— 
9/30/09,  $1,570,000. 

2.  Principal  Investigator,  National  Science  Foundation  MSPA-MCS:  New  Tools 
for  Algebro-Geometric  Data  Analysis,  PI,  $500,000,  8/15/2004-7/31/2007. 

3.  Principal  Investigator,  Mutliscale  Analysis  of  Vegetation  Disease, 

United  States  Forest  Service,  8/16/04-7/31/05,  $15,000. 

4.  Principal  Investigator,  Face  Manifolds  for  Recognition ,  Directorate  of  Central 
Intelligence,  NMA501-03-BAA-0001 

5.  CO-Principal  Investigator,  Geometric  Pattern  Analysis  and  Mental  Task  De¬ 
sign  for  a  Brain- Computer  Interface,  NSF  IIS-0208958 

7.  Patents  and  Disclosures.  The  PI  was  involved  with  a  number  of  patents 
and  disclosures  over  the  courses  of  this  project  including: 

1.  M.  Kirby  and  Yue  Qiao  (2004),  Apparatus,  System  and  Method  for  Inter¬ 
polating  High- Dimensional  Non-linear  Data,  Patent  application  IBM  Docket 
No.:BLD920040032USl 

2.  Michael  Kirby,  Jen-Mei  Chang,  Holger  Kley,  Chris  Peterson,  J.R.  Beveridge 
and  Bruce  Draper,  Set-to-Set  Pattern  Classification,  Patent  filed  8/2007. 
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3.  Michael  Kirby  and  Arta  Jamshidi,  Nonlinear  Function  Approximation  over 
High  Dimensional  Domains ,  Patent  filed  9/2007. 

4.  M.  Kirby  and  Yue  Qiao  (2004),  Apparatus,  System  and  Method  for  Interpo¬ 
lating  High- Dimensional  Non-linear  Data ,  U.S.  Patent  Serial  No.  11/263,740 
Nov.  1,  2005 

5.  M.  Kirby  and  A.  Jamshidi,  A  New  Method  for  Automatic  Order  Determina¬ 
tion  of  Radial  Basis  Function  Models ,  U.S.  Provisional  Patent  Application, 
Express  Mail  EQ  982753865 

6.  M.  Kirby,  Adaptive  Nonlinear  Compression  Algorithm  for  Modeling  and  Pre¬ 
diction ,  U.S.  Provisional  Patent  Application,  Express  Mail  EQ  982753874 

7.  M.  Kirby,  R.  Beveridge,  V.  Chang,  B.  Draper,  H.  Kley  and  C.  Peterson,  A  Low 
Resolution  Grassmannian  Camera  for  Object  Recognition ,  CSURF  disclosure 
06-036,  5/2006. 

8.  M.  Kirby  and  C.  Peterson,  Super  High  Resolution  Biometric  Algorithms , 
CSURF  Disclosure  8/06. 

9.  M  Kirby  and  D.  Dreisigmeyer,  Multivariate  State  Estimation  Technique  for 
pattern  recognition  and  multi-class  image  classification.  2/07 

8.  Honors. 

•  Nominated  for  Preston  Davis  Award  for  Instructional  Innovation. 

•  Nominated  for  College  of  Natural  Sciences  Professor  Laureate. 
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